DOCUMENT RESUME 



ED 390 288 



FL 023 491 



AUTHOR 

TITLE 

PUB DATE 
NOTE 

PUB TYPE 



Martinez, Elizabeth A.; Godev, Concepcion B. 

Should Reading Comprehension Be Tested in the Target 
or the Native Language? A Pilot Study. 

[94] 

26p. 

Reports - Research/Technical (143) — 
Tests/Evaluation Instruments (160) 



EDRS PRICE MF01/PC02 Plus Postage. 

DESCRIPTORS College Students; English; Higher Education; 

’"'Language Tests; ’"'Questioning Techniques; ’'Reading 
Comprehension; ’"'Reading Tests; Second Language 
Instruction; ’"'Second Languages; ’"^Spanish; Test 
Construct i on 



ABSTRACT 

This study investigated the effect of the language in 
which a test is administered when assessing second language reading 
comprehension. Subjects were college students (ages 18-21) enrolled 
in first-semester (n=27) and third-semester (n=l9) Spanish classes . 
Each group was given a different passage in Spanish to read. Two sets 
of questions were distributed. The first was in Spanish and requested 
answers in Spanish and English. The second, distributed after the 
first was collected, was in English and requested answers in English 
and Spanish. The passages and questions are appended. Results suggest 
that students* errors may be caused by factors other than reading 
comprehension of the passage, particularly confusion created by the 
wording of a question or inability to create responses in the target 
language. It is concluded that use of the target language for testing 
decreases the test validity. Improvements in research design are 
suggested. (MSE) 
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Elizabeth A. Martinez 
Concepcion B. Godev 

This paper reports on the results of a pilot study conducted to determine 
the effect exerted by the language chosen when testing reading comprehension. 
College students enrolled in first and third semester Spanish participated in this 
study. The materials used consisted of a reading comprehension paragraph 
followed successively by a set of Spanish questions first and then a set of English 
questions. The subjects were asked to respond to each set of questions in both 
English and Spanish. The results suggest that the use of the target language may 
hamper reading comprehension-test validity. This paper provides an account of 
the error percentages originated by three factors: 1) lack of comprehension of 
questions worded in Spanish, 2) lack of ability to respond in Spanish, and 2) the 
mere lack of passage comprehension. 

Introduction 

The concept of validity constitutes one of the underpinnings of the theory 
and practice of test making (Messick, 1988). It provides a ruler against which 
testing procedures and materials can be measured to determine whether or not a 
given test is in fact measuring what it is supposed to measure. The present work is 
concerned with construct validity, in particular with construct validity of L2 
reading comprehension tests. According to Hughes (1989, p. 26), “a test, part of 
a test, or testing technique is said to have construct validity if it can be 
demonstrated that it measures just the ability which it is supposed to measure”. In 
the case of L2 reading comprehension tests, both type of task (Wolf, 1993; 
Shohamy, 1984) and language of assessment (LI as opposed to L2 and viceversa) 
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(Wolf, 1993b; Lee, 1990, 1987; Hock & Poh, 1979) have been pointed out as 
sources that may hinder construct validity. The language chosen in L2 reading 
comprehension tests has been the focus of attention of only a few studies. These 
works suggest that, all across the different levels of L2 reading competence, the 
language chosen to word comprehension questions and to answer open-ended 
questions may indeed give way to misleading test results. Moreover, whether it is 
LI or L2, how language i- >.sed in wording questions may hinder construct validity 
of a reading comprehension test as well. The wording of a question may reveal the 
answer by making it easy for the test-taker to match words in the questions to 
words in the passage when the questions are formulated in L2. Questions in LI 
may also present the same problem when cognates or semicognates are used, or 
when a reference to a proper noun or date makes it obvious where the answer is 
located in the passage. As for test-takers, they may find some of the test items 
impossible to answer due to their difficulty to i nderstand L2 wording, and at times 
they will find it very challenging to answer a question due to the lack of active 
vocabulary and morpho -syntactic structures (Hock & Poh, 1979; Wolf, 1993b; 
Stroebe, 1930). Finally, as far as the test grader is concerned, answers in L2 may 
turn out so poorly constructed that how to grade them will constitute a real 
dilemma (Stroebe, 1930). 

The preoccupation for lack of construct validity stemming from the 
language chosen in reading comprehension tests does not seem yet to have 
attracted enough attention among publishers or language teachers. Therefore one 
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of the objectives of the present pilot study is to provide more evidence to stir up 
and stimulate discussion of an issue that is still far from being resolved. 

Testing reading in a foreign language is a controversial issue. 
Should reading comprehension be tested in the native language or the target 
language? Educators will argue in favor of the latter because they feel that 
students should have as much exposure to the target language as possible. 
However, by using the target language for the entire testing process leaves doubt 
as to whether the students are actually comprehending the passage; or whether the 
mere writing in the target language hampers conveyance of the message. The 
outcome may be the result of a faulty test design and not the result of faulty 
reading comprehension. Those who argue for a target language-only test claim that 
interference may occur if the student is given a test in the target language and 
required to produce responses in the native language (L. Hart-Gonzalez, personal 
communication, 1994). The purpose of this pilot study is to explore the outcomes 
triggered by different reading-comprehension-test conditions. The objective of the 
study is twofold. First, it was designed to test four hypotheses: How reading 
comprehension is affected when: a.) the question and answer are in the target 
language, i.e., Spanish; b.) the question is in the target language and the answer is 
in the native language, i.e., English; c.) the question and answer are in the native 
language; and d.) the question is in the native language and the answer is in the 
target language. Second, the study was designed to determine whether the 
instruments to test the hypotheses were valid, i.e., the tj^pes of questions and the • 
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level of difficulty of the passage. 

Methods 

Subjects 

Two groups of undergraduate students from George Mason University, 
in Virginia, were selected to participate; 27 were enrolled in &st semester college 
Spanish, and 1 9 in third semester college Spanish. The ages of the subjects ranged 
between eighteen and twenty-one. 

Materials and Procedures 

Each group of students was given a passage to read that varied in length; 
the first s mester passage was 179 words, and the third semester passage was 214 
words. Two sets of questions for each passage were distributed to all students. 
The first set was in Spanish requesting answers in both Spanish and English. The 
second set was in English requesting answers in English and Spanish. This set, 
however, was distributed only after the first set was collected to avoid copying or 
looking back at responses. The first semester and the third semester students were 
asked to respond to eight and nine questions respectively. Below each question 
sufficient space was provided for students to write their answers. The test was 
administered during a regularly scheduled class period of approximately fifty 
minutes. The data was collected during the Spring semester of 1994. See 
Appendices A-F for the passages and questions. 
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Data Analysis 



1. Questions-in-Spanish analyst -j 

The comprehension level of questions formulated in Spanish was 
examined to determine what questions presented difficulties. Whenever a Spanish 
question w'as not answered or was answered incorrectly, whereas its English 
counterpart was answered correctly, it was inferred that the subject could not 
answer the question because she or he did not understand it. Non-understood 
questions were classified by the number assigned in the question sheet for reading 
comprehension and by the number of subjects who failed to understand a given 
question. 

2. Answers-in-Spanish analysis 

Spanish answers were classified accord.ing to the number assigned to the 
questions to which they responded. Answers were further categorized into 
incorrect answers triggered by the subject’s lack of target language writing skills, 
and incorrect answers triggered by faulty reading comprehension of the passage. 

To determine the cause of subjects’ incorrect answers, English and 
Spanish responses were compared. An answer was considered incorrect if it did 
not address the qaestion or if the answer was not comprehensible. Blanks or 
incorrect answers triggered by a misunderstood question were left out in this phase 
since they were examined under Questions-in-Spanish Analysis. 
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Results 

First semester reading comprehension test 

1. Questions in Spanish 

It was observed that out of a total of eight questions, four- (50%) 
questions were not understood by at least one subject. A total of nine (33%) 
subjects did not understand at least one of the questions. The mean percentage of 
non-understood questions was 4% (see Table 1 and graph). 

2. Answers in Spanish 

Sbc (75%) answers out of eight were not correctly answered at least once 
due to the lack of writing skills in Spanish, and sbc (75%) answers out of eight 
were not answered correctly at least once as a result of faulty reading 
comprehension. A total of 16 (59%) subjects were not able to answer at least one 
question, one (3%) subject was not able to answer two (22%) questions, and two 
(7%) subjects were not able to answer three (33%) questions as a result of 
insufficient writing skills. The mean percentage of erroneous answers generated by 
lack of writing skills was 13.4% (see graph). The breakdown of erroneous 
responses generated by limited reading comprehension is as follows: A total of 
nine (33 %) subjects did not respond correctly to at least one (3%) question; and 
three (1 1%) did not respond correctly to at least two (22%) questions. The mean 
number of erroneous answers due to the lack of reading comprehension was 7.9% 
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(see Table 2 and graph). 
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Third semester reading comprehension test 

1. Questions in Spanish 

A total of six (66%) questions out of nine were not understood by at 
least one subject. A total of eight (42%) subjects did not understand at least one 
of the questions, one (five percent) subject did not understand five (55%) 
questions, one (five percent) subject did not understand three (33%) questions, and 
one (5%) subject did not understand two (22%) questions. The mean percentage 
of non-understood questions was 8.7% (see Table 1 and graph). 

2. Answers in Spanish 

Seven (77%) answers out of nine were not correctly answered at least once 
due to the lack of writing skills in Spanish, and nine (100%) answers out of nine 
were not answered correctly at least once as a result of faulty reading 
comprehension. A total of sbc (31%) subjects were not able to answer at least one 
(11%) question, one (5%) subject was not able to answer two (22%) questions, 
and two (10%) did not answer three (33%) questions as a result of limited writing 
skills. The mean percentage of erroneous answers generated by lack of writing 
skills was 8.1% (see graph). The breakdown of erroneous responses generated by 
limited reading comprehension is as follows: A total of four (21%) subjects did 
not respond correctly to at least one (1 1%) question; five (26%) subjects did not 
respond correctly to two (22%) questions; four (21%) subjects did not answer 
correctly to three (33%) questions; one (1 1%) subject did not answer correctly to 
four (44%) questions, two (10%) did not answer correctly to five (55%) 
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questions; and two (10%) subjects did not answer correctly to sbc (66%) 
questions. The mean number of erroneous answers due to lack of reading 
comprehension was 30.68% (see Table 2 and graph). 
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Table 1 

Questions in Spanish 
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Groups Mean Percentages 




■^Percentage of non-understood-questions occurrences in the group. 

♦♦Percentage of erroneous answers yielded by lack of target-language writing skills. 
♦♦♦Percentage of erroneous answers yielded by lack of comprehension of the reading 
passage. 
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Discussion 

The questions formulated in the target language were a source of error in 

the responses of both first and third semester Spanish students. Likewise, their 

answers in the target language were an additional source of error. However, errors 

generated by the lack of writing skills were more frequent in the first semester 

* 

group than in the third semester. The third semester group performed better as a 
result of both more experience with the language and with the management of 
strategies to use the very passage to shape their answers. 

Apart from such observations of the lack of writing skills and the 
incomprehensibility of the questions in the target language, other observations 
were made that suggested inaccuracies in the students' comprehension of the 
passage. 

The first semester group responded in the target language to questions 
she (Q6) and eight (Q8), (see appendices), without discriminating how much of the 
sentence from the p?issage was needed to respond. This was obvious when 18 
subjects ended their responses to Q6 Why does Luis travel constantly?, and Q8 
What is a tour guide?, with>^ ademds ‘and also’. This indicates that the students 
copied directly from the passage without comprehending what they were wTiting. 
The students, through vocabulary fi'om the question, were able to locate the 
answer. However, their response proves that comprehension was not fully 
obtained. 
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The third semester group indicated on occasion literal translation. With the 
responses to Q4, one example of this was found by two of the subjects. In their 
first set, i.e., Spanish Question; iDonde conocio Martin a Rosario?, both of these 
subjects responded in English in the hospital. The actual passage reads in Spanish; 
en el hospital. However when these same students were given the second set, i.e., 
English Question; Where did Martin meet Rosario?, each responded at the 
hospital. It appears that the question in the target language may not have been 
fully understood, but it provided enough vocabulary from the passage for the 
students to search for the response. The students were able to locate the answer in 
the passage through the Spanish question’s conocio and literally translated their 
response, which changes when given the English question. 

In comparing the proportion of first and third semester incorrect answers 
generated by faulty comprehension of the passage, observations show more errors 
in the third semester group than in the first semester group. To determine whether 
the discrepancy of results was due to excessive passage difficulty or easiness, a 
distribution analysis was conducted on the frequency of incorrect answers 
generated by lack of reading comprehension. The results showed that the 
distribution of incorrect answers was normal in the third semester group, which 
shows that the difficulty level of the passage was fair; however the distribution of 
incorrect answers in the first semester group was skewed to the right, which means 
that the passage was very easy. This result, although unexpected, shows soundly 
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that students might understand the passage and still display a performance 
inconsistent with their level of reading comprehension. 

Some of the answers, especially with those where the question was in 
English, were left blank because of possible emotional factors, or were memorized 
from what was written on the test with the Spanish question. For examph; in the 
third semester group, one of the subject’s response to Q8 was Una persona de 
vender, porque vender los productos de su compama, which is word for word the 
exact same answer provided on the English question test; since the English 
question test was identical to that of the Spanish, this is not unlikely. Although the 
Spanish question test was collected before administering the English set of 
questions, some subjects seemed to have been able to remember their responses. 

Two emotional levels, boredom and anxiety, were considered whenever a 
student would respond correctly to the first set of questions (the most challenging 
since they were formulated in Spanish), and would leave the same question blank 
or partially blank in the second set. Boredom may have been the outcome of 
having to carry out a nearly similar task twice. The students may have felt 
discouraged to perform their best when they realized that the second set of 
questions did not represent a challenge. Test anxiety may have been responsible for 
the cases of students who understood a question in Spanish and showed a lack of 
passage comprehension in their answer to the question formulated in Spanish, but 
showed comprehension in the answer to the question formulated in English and 



vice versa. 
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Conclusion 

The results of this study show that students' errors on a reading 
comprehension test may be caused by factors other than reading comprehension of 
the passage. In combining error percentages it is apparent that 68% of the errors 
of the first semester group and 35% from the third semester group stemmed from 
the combined factors of not understanding the wording of the questions and firom 
not being able to word the answers in Spanish. Moreover, observations show that 
students may be able to answer correctly in Spanish without having full 
understanding of the passage, as explained with examples under the section of 
Discussion. Some students develop strategies to locate the lines where the answer 
may likely be by detecting key words fi'om the question. This is especially true of 
third semester students, who have had more experience with the language and 
language tests thus more time to develop certain test strategies. However, in this 
study, this phenomenon also occurred with the first semester students, although 
the result was still an incorrect answer, as demonstrated in the Discussion section. 

It is apparent in these results that the use of Spanish to assess reading 
comprehension decreases the test validity since students give the impression they 
do understand when in fact they do not and vice versa. This pilot study has clearly 
illustrated that other means of testing reading comprehension may be considered to 
ensure test validity. 

Should this study be replicated on a larger scale, several changes on the 
research design should be implemented: 
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1. ) The passage for the first semester group should increase its level of 

difficulty by including a wider range of vocabulary, more complex sentences, and 
more words. The reading comprehension questions should not include key words 
from the passage that might cue students to the location of the answer, resulting in 
copying the passage. 

2. ) To control for boredom and frustration, the experiment should be set up so 

that there is no repetition of the task, i.e., Spanish questions with Spanish and 
English responses, and English questions with English and Spanish responses. To 
avoid this, the test would be administered to four different groups per level. The 
scheme to control for language variable would be as follows: 





Grout) A 


Group B 


Group C 


Group E 


Question 


Spanish 


English 


Spanish 


English 


Answer 


Spanish 


English 


English 


Spanish 



All levels should follow the same scheme. 

3.) A sociolinguistic questionnaire should be set up to include questions 
dealing with background language training, interest in the subject, and anxiety. 
This instrument would allow to control for previous knowledge effect, motivation 
and anxiety. 

The replication of this study, including the changes discussed in this 
section, may enlighten educators to search for new methods of assessing reading 
comprehension in foreign languages. 
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Notes 

1. This passage is an adapted version of a passage in Van Patten et al. (1991, 19) 

2. This passage is an adapted version of a passage in Van Patten et al. 

(1991, 101) 
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Appendix A 



First semester passage 



La familia Ramirez' 

Elena Ramirez y Luis son los padres de Miguel y Jaime. Elena Ramirez es 
la madre de Miguel y Jaime. Ella no trabaja fiiera de casa porque con dos hijos 
Elena tiene ya mucho trabajo. Jaime, especialmente, le da muchos problemas. 

Jaime es un nino con mucha energia. Es el menor de los dos hijos y le 
gusta ser el centre de atencion. No es muy buen estudiante y Elena tiene que 
hablar con frecuencia con su profesor y el director de su escuela. Este ano Jaime 
tiene muchas dificultades en la clase de matematicas. 

AI contrario de Jaime, Miguel es un hijo modelo y Elena esta muy 
orgullosa de el. Miguel es inteligente, estudioso y sus profesores hablan muy bien 
de el. 

Luis, el esposo de Elena, pasa mucho tiempo fliera de casa por causa de su 
trabajo. El es guia turistico por eso tiene que viajar constantemente y ademas no 
tiene un horario fijo. Generalmente, el ayuda a Jaime y Miguel con las tareas de la 
clase de historia porque el es un experto en esta materia. 
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Appendix B 



First semester Spanish questions 



After reading the paragraph, ANSWER the questions in SPANISH and in 
ENGLISH. 

1 . ^Como se llaman los padres de Miguel y Jaime? 

SPANISH: 

ENGLISH: 

2. ;Por que no trabaia Elena fuera de casa? 

SPANISH: 

ENGLISH: 

3. ^Como es Jaime? 

SPANISH: 

ENGLISH: 

4. ^Por que Elena tiene que hablar con el profesor de Jaime con frecuencia? 

SPANISH: 

ENGLISH: 

5. ^De quien esta Elena muy orgullosa? ^Por que? 

SPANISH: 

ENGLISH: 

6. ^Por que tiene qur viajar Luis constantemente? 

SPANISH: 

ENGLISH: 

7. Compara las caracteristicas de Jaime y Miguel. 

SPANISH: 

ENGLISH: 

8. Define “guia turistico”. 

SPANISH: 

ENGLISH: 
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Appendix C 



First semester English questions 



After reading the paragraph, ANSWER the questions in ENGLISH and in 
SPANISH. 

1 . What are Miguel and Jaime’s parents’ names? 

ENGLISH: 

SPANISH: 

2. Why does Elena not work outside the home? 

ENGLISH: 

SPANISH: 

3. What is Jaime like? 

ENGLISH: 

SPANISH: 

4. Why does Elena have to fi-equently speak with Jaime’s teacher? 

ENGLISH: 

SPANISH: 

5. Who is Elena proud of? Why? 

ENGLISH: 

SPANISH: 

6. Why does Luis have to constantly travel? 

ENGLISH: 

SPANISH: 

7. Compare Jaime and Miguel’s personalities? 

ENGLISH: 

SPANISH: 

8 . Define “guia turistico 

ENGLISH: 

SPANISH: 
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Appendix D 



Third semester passage 



Martin Iglesias" 

Martin Iglesias tenia una vida prospera en su pais, la Argentina; pero 
decidio ir a Espana porque era un hombre aventurero y porque penso que Espana 
ofrecia una buena oportunidad para vender los productos de su compania. For 
esta razon flie primero a Madrid y luego a Barcelona. 

Martin tenia otro motive para viajar a Espana, queria buscar a una hermana 
de su madre que vivia en Sevilla y con la que habia perdido el contacto. Solo tenia 
una vieja direccion y el nombre del hospital en que su tia trabajaba. Llego a la casa 
de la direccion pero nadie conocia a esa senora. Uno de los vecinos le dio la 
direccion del hospital. En el hospital Martin conocio a una mujer llamada Rosario. 

Rosario era viuda, su marido habia muerto hacia tres anos, y tenia un hijo 
de cinco anos. Ella era abogada y le gustaba mucho su trabajo. 

Aunque Rosario tenia siempre mucho trabajo, ella ayudo a Martin a buscar 
a la hermana de su madre. Finalmente descubrieron que la tia de Martin habia 
muerto hacia dos anos y que los hijos de su tia vivian ahora en el sur de Francia. 

Ahora Martin tenia que volver a la Argentina, pero no queria irse sin' 
Rosario porque se habia enamorado de ella. 
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Appendix £ 



Third semester Spanish questions 



After reading the paragraph, ANSWER the questions in SPANISH and in 
ENGLISH. 

1 . De tres razones de por que Martin decidio irse a Espana. 

SPANISH; 

ENGLISH: 

2. ^Por que se ftie Martin a Madrid y a Barcelona? 

SPANISH: 

ENGLISH; 

3. ^Por que se ftie Martin a Sevilla? 

SPANISH: 

ENGLISH; 

4. ^Donde conocio Martin a Rosario? 

SPANISH; 

ENGLISH; 

5. ^Cual es el trabajo de Rosario? 

SPANISH: 

ENGLISH; 

6. ^Cual es el trabajo de Martin? Justiiique su respuesta. 

SPANISH; 

ENGLISH; 

7. ^Por que no pudo Martin encontrar a su tia? 

SPANISH; 

ENGLISH; ^ 

8. ^Como supo Martin la direccion del hospital? 

SPANISH: 

ENGLISH: 

9. Define la palabra “viuda'’. 

SPANISH; 

ENGLISH: 
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Appendix F 



Third semester English questions 



After reading the paragraph, ANSWER the questions in ENGLISH and in 
SPANISH. 

1 . Give three reasons why Martin decided to go to Spain. 

ENGLISH: 

SPANISH: 

2. Why did Martin go to Madrid and Barcelona? 

ENGLISH: 

SPANISH: 

3. Why did Martin go to Seville? 

ENGLISH: ^ 

SPANISH: 

4. Where did Martin meet Rosario? 

ENGLISH: 

SPANISH: 

5. What is Rosario’s job? 

ENGLISH: 

SPANISH: 

6. What is Martin’s job? Justify your response. 

ENGLISH: 

SPANISH: 

7. Why couldn’t Martin find his aunt? 

ENGLISH: 

SPANISH: 

8. How did Martin find out the address to the hospital? 

ENGLISH: 

SPANISH: 

9. Defiile the word “viuda.” 

ENGLISH: 

SPANISH: 



